{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# -*- coding: utf-8 -*-\n",
    "# @author: tongzi\n",
    "# @created date: 2019/10/19\n",
    "# @description: kernel density estimation\n",
    "# @last modification: 2019/10/19"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#import libraries\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们之前提过，密度评估器是一种利用 D 维数据集生成 D 维概率分布估计的算法。GMM 算法用不同高斯分布的加权汇总来表示概率分布估计。核密度估计（kernel density estimation，KDE）算法将高斯混合理念扩展到了逻辑极限（logical extreme）。它通过对每个点生成高斯分布的混合成分，获得本质上是无参数的密度评估器。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.13.1 KDE的由来：直方图  \n",
    "密度评估器是一种寻找数据集生成概率分布模型的算法。你可能很熟悉一维数据的密度估计——直方图，那是一个简单的密度评估器。直方图将数据分成若干区间，统计落入每个区间内的点的数量，然后用直观的方式将结果可视化。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面创建两组服从正态分布的数据："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_data(N, f=0.3, rseed=1):\n",
    "    rng = np.random.RandomState(rseed)\n",
    "    x = rng.randn(N)\n",
    "    x[int(f * N):] += 5\n",
    "    return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = make_data(1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "前面已经介绍过，基于计数的标准直方图可以用 plt.hist() 函数来生成。只要确定直方图的 normed 参数，就可以得到一个正态分布直方图。在这个直方图中，区间的高度并不反映统计频次，而是反映概率密度:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# bins表示分成30个区间，normed=True标准化数据\n",
    "hist = plt.hist(x, bins=30, normed=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在区间不变的条件下，这个标准化（计算概率密度）只是简单地改变了 y\n",
    "轴的比例，相对高度仍然与频次直方图一致。标准化是为了让直方图的总面积等于 1，可\n",
    "以通过检查直方图函数的输出结果来确认这一点："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "density, bins, patches = hist\n",
    "widths = bins[1:] - bins[:-1]\n",
    "(density * widths).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用直方图作为密度评估器时需要注意的是，区间大小和位置的选择不同，产生的统计特\n",
    "征也不同。例如，如果只看数据中的 20 个点，选择不同的区间将会出现完全不同的解读\n",
    "方式（直方图），如以下示例:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = make_data(20)\n",
    "bins = np.linspace(-5, 10, 10) # 10 bins\n",
    "fig, axes = plt.subplots(1, 2, figsize=(12, 4), sharex=True,\n",
    "                        sharey=True, subplot_kw={\n",
    "                            'xlim': (-4, 9),\n",
    "                            'ylim': (-0.02, 0.3)\n",
    "                        })\n",
    "fig.subplots_adjust(wspace=0.05)\n",
    "for i, offset in enumerate([0.0, 0.6]):\n",
    "    # note: the argument 'normed' is deprecated, using 'density' instead.\n",
    "    axes[i].hist(x, bins=bins+offset, density=True)\n",
    "    axes[i].plot(x, np.full_like(x, -0.01), '|k', markeredgewidth=1)\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "左边的直方图很清楚地显示了一个双峰分布，但右边的图却显示了一个单峰分布，并带着\n",
    "一个长尾。如果没有看到前面的代码，你可能会认为这两个直方图描述的是不同的数据。\n",
    "这样怎么能相信直方图的可视化结果呢？如何才能改进这个问题呢？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots()\n",
    "bins = np.arange(-3 ,8)\n",
    "ax.plot(x, np.full_like(x, -0.1), '|k', markeredgewidth=1)\n",
    "# np.histogram()返回值：\n",
    "# hist：每个区间的频数； bin_edges：区间的边缘\n",
    "# \n",
    "for count, edge in zip(*np.histogram(x, bins)):\n",
    "  for i in range(count):\n",
    "    # plt.Rectangle(xy, width, height,...):\n",
    "    # xy:矩形左下角的坐标；width：矩形的宽度； height：矩形的高度\n",
    "    ax.add_patch(plt.Rectangle((edge, i), 1, 1, alpha=0.5))\n",
    "    \n",
    "ax.set_xlim(-4, 8)\n",
    "ax.set_ylim(-0.2, 8)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_d = np.linspace(-4, 8, 2000)\n",
    "density = sum((abs(xi - x_d) < 0.5 for xi in x))\n",
    "plt.fill_between(x_d, density, alpha=0.5)\n",
    "plt.plot(x, np.full_like(x, -0.1), '|k', markeredgewidth=1)\n",
    "plt.axis([-4, 8, -0.2, 8])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "虽然结果看起来有点杂乱，但与标准直方图相比，这幅图可以更全面地反映出数据的真实\n",
    "特征。不过，这些粗糙的线条既没有让人愉悦的美感，也不能反映数据的任何真实性质。\n",
    "为了让线条显得更加平滑，我们也许可以用平滑函数取代每个位置上的方块，例如使用高\n",
    "斯函数。下面用标准的正态分布曲线代替每个点的方块:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import norm\n",
    "x_d = np.linspace(-4, 8, 1000)\n",
    "# norm是一个类，其参数：\n",
    "# loc: the mean\n",
    "# scale: the standard deviation\n",
    "density = sum(norm(loc=xi).pdf(x_d) for xi in x)\n",
    "\n",
    "plt.fill_between(x_d, density, alpha=0.5)\n",
    "plt.plot(x, np.full_like(x, -0.1), '|k', markeredgewidth=1)\n",
    "plt.axis([-4, 8, -0.2, 5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这幅平滑的图像是由每个点所在位置的高斯分布构成的，这样可以更准确地表现数据分布\n",
    "的形状，并且拟合方差更小（也就是说，进行不同的抽样时，数据的改变更小）。\n",
    "最后两幅图是核密度估计在一维中的示例，第一幅图用的是“tophat”核，第二幅图用的\n",
    "是高斯核。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.13.2核密度估计的实际应用  \n",
    "核密度估计的自由参数是核类型（kernel）参数，它可以指定每个点核密度分布的形状。\n",
    "而核带宽（kernel bandwidth）参数控制每个点的核的大小。在实际应用中，有很多核可用于核密度估计，特别是 Scikit-Learn 的 KDE 实现支持六个核，详情请参见 Scikit-Learn 的核密度估计文档（http://scikit-learn.org/stable/modules/density.html）。  \n",
    "  \n",
    "虽然 Python 中有许多核密度估计算法实现的版本（特别是在 SciPy 和 StatsModels 包中），但是我更倾向于使用 Scikit-Learn 的版本，因为它更高效、更灵活，在 sklearn.neighbors.KernelDensity 评估器中实现，借助六个核中的任意一个核、二三十个距离量度就可以处理具有多个维度的 KDE。由于 KDE 计算量非常大，因此 Scikit-Learn 评估器底层使用了一种基于树的算法，可以利用 atol （绝对容错）和 rtol （相对容错）参数来平衡计算时间与准确性。我们可以用 Scikit-Learn 的标准交叉检验工具来确定自由参数核带宽，后面将会介绍这些内容。  \n",
    "下面先来看一个简单的示例，利用 Scikit-Learn 的 KernelDensity 评估器来重现前面的图像："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KernelDensity\n",
    "\n",
    "kde = KernelDensity(bandwidth=1.0, kernel='gaussian')\n",
    "kde.fit(x[:, None]) # or kde.fit(x[:, np.newaxis])\n",
    "\n",
    "# return the logarithm of probability density\n",
    "logprob = kde.score_samples(x_d[:, None])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.fill_between(x_d, np.exp(logprob), alpha=0.5)\n",
    "plt.plot(x, np.full_like(x, -0.01), '|k', markeredgewidth=1)\n",
    "plt.ylim(-0.02, 0.22)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里的结果经过了标准化处理，保证曲线下的面积为 1。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 通过交叉检验选择带宽  \n",
    "在 KDE 中，带宽的选择不仅对找到合适的密度估计非常重要，也是在密度估计中控制偏\n",
    "差-方差平衡的关键：带宽过窄将导致估计呈现高方差（即过拟合），而且每个点的出现\n",
    "或缺失都会引起很大的不同；带宽过宽将导致估计呈现高偏差（即欠拟合），而且带宽较\n",
    "大的核还会破坏数据结构。  \n",
    "  \n",
    "关于如何利用统计方法快速对数据作出严格假设，从而确定密度估计的最佳带宽的研究\n",
    "由来已久：如果你查看 SciPy 和 StatsModels 包中 KDE 的实现，就能看到这些规则的实现过程。  \n",
    "  \n",
    "前面介绍过，机器学习中超参数的调优通常都是通过交叉检验完成的。Scikit-Learn 团队在设计 KernelDensity 评估器时，就设想了直接使用 Scikit-Learn 的标准网格搜索工具。这里用 GridSearchCV 来优化前面数据集的密度估计带宽。因为我们要处理的数据集规模比较小，所以使用留一法进行交叉检验，该方法在每一次做交叉检验时，都会最小化训练集的损失："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import LeaveOneOut\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bandwidths = 10 ** np.linspace(-1, 1, 100)\n",
    "grid = GridSearchCV(KernelDensity(kernel='gaussian'),\n",
    "                   {'bandwidth': bandwidths},\n",
    "                   cv=LeaveOneOut())\n",
    "grid.fit(x[:, None]);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grid.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这个最优带宽与前面示例图像中使用的带宽非常接近，那里使用的带宽是 1.0（也是\n",
    "scipy.stats.norm 的默认宽度）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.13.3例子：球形空间的KDE  \n",
    "KDE 可能最常用于描述数据点的分布。例如，在 Seaborn 可视化库中（详情请参见 4.16\n",
    "节），KDE 是内置画图函数，可以自动对一维和二维空间的数据进行可视化。  \n",
    "  \n",
    "下面将介绍一个稍复杂些的 KDE 数据分布可视化应用。我们会使用 Scikit-Learn 的一些地理数据：褐喉树懒（bradypus variegatus）和森林小稻鼠（microryzomys minutus）这两种南美洲哺乳类动物的地理分布观测值。  \n",
    "  \n",
    "可以用 Scikit-Learn 直接获取数据："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_species_distributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading species data from https://ndownloader.figshare.com/files/5976075 to C:\\Users\\John\\scikit_learn_data\n",
      "Downloading coverage data from https://ndownloader.figshare.com/files/5976078 to C:\\Users\\John\\scikit_learn_data\n"
     ]
    }
   ],
   "source": [
    "# note: 该例子无法下载数据\n",
    "data = fetch_species_distributions()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.13.4例子：不是很朴素的贝叶斯  \n",
    "下面这个示例是用 KDE 做贝叶斯生成分类，同时演示如何使用 Scikit-Learn 自定义评\n",
    "估器。我们介绍了朴素贝叶斯分类方法，为每一个类创建了一个简单的生成模型，\n",
    "并用这些模型构建了一个快速的分类器。在朴素贝叶斯分类中，生成模型是与坐标轴平行\n",
    "的高斯分布。如果利用 KDE 这样的核密度估计算法，我们就可以去掉“朴素”的成分，\n",
    "使用更成熟的生成模型描述每一个类。虽然它还是贝叶斯分类，但是它不再朴素。  \n",
    "  \n",
    "一般分类器的生成算法如下所示：  \n",
    "(1) 通过标签分割训练数据。  \n",
    "(2) 为每个集合拟合一个 KDE 来获得数据的生成模型，这样就可以用任意$x$观察值和$y$标签计算出似然估计值$P(x|y)$。  \n",
    "(3) 根据训练集中每一类的样本数量，计算每一类的先验概率$P(y)$。  \n",
    "(4) 对于一个未知的点$x$，每一类的后验概率是$P(x|y)∝P(x| y)P (y)$，而后验概率最大的类就是分配给该点的标签。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.base import BaseEstimator, ClassifierMixin\n",
    "class KDEClassifier(BaseEstimator, ClassifierMixin):\n",
    "    '''\n",
    "    Bayessian classifier based on KDE\n",
    "    \n",
    "    parameters:\n",
    "    ------------\n",
    "    bandwidth: float\n",
    "        bandwidth for each classifier\n",
    "    kernel: str\n",
    "        the name of kernel function, which is passed to KernelDensity\n",
    "    \n",
    "    '''\n",
    "    def __init__(self, bandwidth=1.0, kernel='gaussian'):\n",
    "        self.bandwidth = bandwidth\n",
    "        self.kernel = kernel\n",
    "        \n",
    "    def fit(self, X, y):\n",
    "        try:\n",
    "            self.classes_ = np.sort(np.unique(y))\n",
    "            training_sets = [X[y == yi] for yi in self.classes_]\n",
    "            self.models_ = [KernelDensity(bandwidth=self.bandwidth,\n",
    "                                         kernel=self.kernel).fit(Xi)\n",
    "                           for Xi in training_sets]\n",
    "            assert X.shape[0]\n",
    "            self.logpriors_ = [np.log(Xi.shape[0] / X.shape[0]) \n",
    "                              for Xi in training_sets]\n",
    "        except Exception as e:\n",
    "            print(repr(e))\n",
    "        return self\n",
    "    \n",
    "    def predict_proba(self, X):\n",
    "        logprobs = np.array([model.score_samples(X) \n",
    "                            for model in self.models_]).T\n",
    "        result = np.exp(logprobs + self.logpriors_)\n",
    "        return result / result.sum(1, keepdims=True)\n",
    "    \n",
    "    def predict(self, X):\n",
    "        return self.classes_[np.argmax(self.predict_proba(X), 1)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.解析自定义解析器  \n",
    "在 Scikit-Learn 中，每个评估器都是一个类，这个类可以很方便地继承 BaseEstimator 类（其中包含了各种标准功能），也支持适当的混合类（mixin，多重继承方法）。例如，除了标准功能之外， BaseEstimator 还包含复制一个评估器的必要逻辑，可以用于交叉检验。另外， ClassifierMixin 也定义了一个 score() 方法可以用作交叉检验。后面会提供一个文档字符串，它可以被 IPython 的帮助功能捕获到。  \n",
    "  \n",
    "接下来是类初始化方法：  \n",
    "```python\n",
    "    def __init__(self, bandwidth=1.0, kernel='gaussian'):\n",
    "        self.bandwidth = bandwidth\n",
    "        self.kernel = kernel\n",
    "\n",
    "```  \n",
    "这是 KDEClassifier() 实例化对象时执行的代码。在 Scikit-Learn 中，除了将参数值传递给self 之外，**初始化不包含任何操作**。这点很重要，因为 BaseEstimator 的逻辑需要满足用于交叉检验、网格搜索和其他功能的评估器的复制和修改需求。同样， $__init__$ 中所有的参数都是显式的，不要使用 $*args$ 或 $**kwargs$ ，因为这些可变参数在交叉检验方法中无法被正确理解。  \n",
    "  \n",
    "接下来用 fit() 方法处理训练数据： \n",
    "```python\n",
    "    def fit(self, X, y):\n",
    "        self.classes_ = np.sort(np.unique(y))\n",
    "        training_sets = [X[y == yi] for yi in self.classes_]\n",
    "        self.models_ = [KernelDensity(bandwidth=self.bandwidth,\n",
    "                                     kernel=self.kernel).fit(Xi)\n",
    "                       for Xi in training_sets]\n",
    "        self.logpriors_ = [np.log(Xi.shape[0] / X.shape[0]) \n",
    "                          for Xi in training_sets]\n",
    "        return self\n",
    "```\n",
    "首先在训练数据集中找到所有类（对标签去重），为每一类训练一个 KernelDensity 模型；然后，根据输入样本的数量计算类的先验概率；最后用 fit() 函数返回 self ，这么做可以串联所有命令，例如：  \n",
    "```python\n",
    "   label = model.fit(X, y).predict(X)\n",
    "```  \n",
    "请注意，所有拟合结果都将存储在一个带后下划线的变量中（例如 self.logpriors_ ）。这一招在 Scikit-Learn 中很好使，你可以快速浏览一个评估器的所有拟合结果（利用 IPython的 Tab 自动补全），并且查看哪个结果与训练数据的拟合最好。  \n",
    "  \n",
    "终于，我们得到了对新数据进行预测的逻辑：  \n",
    "```python\n",
    "    def predict_proba(self, X):\n",
    "        logprobs = np.array([model.score_samples(X) \n",
    "                                for model in self.models_]).T\n",
    "        result = np.exp(logprobs + self.logpriors_)\n",
    "        return result / result.sum(axis=1, keepdims=True)\n",
    "    \n",
    "    def predict(self, X):\n",
    "        return self.classes_[np.argmax(self.predict_proba(X), axis=1)]\n",
    "\n",
    "```  \n",
    "因为这是一个概率分类器，所以首先实现 predict_proba() ，它返回的是每个类概率的数组，数组的形状为 [n_samples, n_classes] 。这个数组中的 [i, j] 表示样本 i 属于 j 类的后验概率，用似然估计先乘以类先验概率，再进行归一化。  \n",
    "  \n",
    "最后， predict() 函数根据这些概率，返回概率值最大的类。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.使用自定义评估器  \n",
    "尝试用自定义评估器解决前面介绍过的一个问题：手写数字的分类问题。首先导入手\n",
    "写数字，然后对一个带宽范围内的数据使用 GridSearchCV 元评估器（详情请参见 5.3 节），从而计算交叉检验值："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_digits\n",
    "from sklearn.model_selection import GridSearchCV\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits = load_digits()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "bandwiths = 10 ** (np.linspace(0, 3, 100))\n",
    "grid = GridSearchCV(KDEClassifier(), {'bandwidth': bandwidths})\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\Anaconda3\\lib\\site-packages\\sklearn\\model_selection\\_split.py:1978: FutureWarning: The default value of cv will change from 3 to 5 in version 0.22. Specify it explicitly to silence this warning.\n",
      "  warnings.warn(CV_WARNING, FutureWarning)\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n",
      "D:\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:36: RuntimeWarning: invalid value encountered in true_divide\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv='warn', error_score='raise-deprecating',\n",
       "             estimator=KDEClassifier(bandwidth=1.0, kernel='gaussian'),\n",
       "             iid='warn', n_jobs=None,\n",
       "             param_grid={'bandwidth': array([ 0.1       ,  0.10476158,  0.10974988,  0.1149757 ,  0.12045035,\n",
       "        0.12618569,  0.13219411,  0.13848864,  0.14508288,  0.15199111,\n",
       "        0.15922828,  0.16681005,  0.17475284,  0.18307383,  0.19179103,\n",
       "        0.2009233 ,  0.21049041,  0.2205...\n",
       "        3.27454916,  3.43046929,  3.59381366,  3.76493581,  3.94420606,\n",
       "        4.1320124 ,  4.32876128,  4.53487851,  4.75081016,  4.97702356,\n",
       "        5.21400829,  5.46227722,  5.72236766,  5.9948425 ,  6.28029144,\n",
       "        6.57933225,  6.8926121 ,  7.22080902,  7.56463328,  7.92482898,\n",
       "        8.30217568,  8.69749003,  9.11162756,  9.54548457, 10.        ])},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring=None, verbose=0)"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid.fit(digits.data, digits.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['mean_fit_time', 'std_fit_time', 'mean_score_time', 'std_score_time', 'param_bandwidth', 'params', 'split0_test_score', 'split1_test_score', 'split2_test_score', 'mean_test_score', 'std_test_score', 'rank_test_score'])"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid.cv_results_.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.11407902, 0.11407902, 0.11407902, 0.11407902, 0.10740122,\n",
       "       0.17028381, 0.11240957, 0.11630495, 0.20033389, 0.11519199,\n",
       "       0.16583194, 0.18085698, 0.20701169, 0.1836394 , 0.13021703,\n",
       "       0.19476906, 0.17751809, 0.11630495, 0.19699499, 0.25431274,\n",
       "       0.20144686, 0.2654424 , 0.25041736, 0.14134669, 0.23094046,\n",
       "       0.2459655 , 0.23372287, 0.26099054, 0.29660545, 0.40456316,\n",
       "       0.31608236, 0.34168058, 0.46744574, 0.3967724 , 0.48469672,\n",
       "       0.53199777, 0.49137451, 0.56649972, 0.56983862, 0.54479688,\n",
       "       0.57762938, 0.5147468 , 0.56037841, 0.51641625, 0.50695604,\n",
       "       0.53811909, 0.52754591, 0.49749583, 0.48024485, 0.4590985 ,\n",
       "       0.53422371, 0.46466333, 0.47356706, 0.57762938, 0.49471341,\n",
       "       0.48414023, 0.47690595, 0.50306066, 0.48580968, 0.52031163,\n",
       "       0.50027824, 0.58319421, 0.59154146, 0.52476349, 0.63550362,\n",
       "       0.639399  , 0.68670006, 0.72398442, 0.78185865, 0.81524763,\n",
       "       0.84474124, 0.86087924, 0.89204229, 0.91541458, 0.92821369,\n",
       "       0.94212577, 0.95381191, 0.95659432, 0.95882026, 0.96271564,\n",
       "       0.9638286 , 0.96438509, 0.96438509, 0.9638286 , 0.9638286 ,\n",
       "       0.96438509, 0.9638286 , 0.9638286 , 0.9638286 , 0.96549805,\n",
       "       0.96549805, 0.96605454, 0.96828047, 0.96605454, 0.9671675 ,\n",
       "       0.96605454, 0.96605454, 0.9638286 , 0.96104619, 0.95659432])"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid.cv_results_['mean_test_score']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x17b0d5994e0>]"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD+CAYAAAAqP/5ZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deXicV2Hv8e+ZGe37almLbdmW13gjsrPvC0mAQCGFpOxNSGkJpYX2Xri03JbeFriltJcWSNMmpGFJCJQlJCEQQuIszmJ5ieNFtmVZtiRrtXZptMzMuX/MYi0ja/FII838Ps/j58m879E7R/Ho56OzGmstIiKy+DmiXQEREYkMBbqISIxQoIuIxAgFuohIjFCgi4jECAW6iEiMmDLQjTEPGWNajTEHJ7lvjDHfNMbUGGMOGGPeFvlqiojIVKbTQn8YuOU8928FKgJ/7gW+c+HVEhGRmXJNVcBa+6IxZsV5irwbeMT6Vyi9ZozJNsYstdY2ne+5+fn5dsWK8z1WRETG27NnT7u1tiDcvSkDfRpKgPpRrxsC184b6CtWrKCqqioCby8iEj+MMacmuxeJQVET5lrY/QSMMfcaY6qMMVVtbW0ReGsREQmKRKA3AGWjXpcCZ8IVtNY+YK2ttNZWFhSE/Y1BRERmKRKB/gTwkcBsl0uB7qn6z0VEJPKm7EM3xjwKXAvkG2MagP8NJABYa+8HngZuA2qAAeDjc1VZERGZ3HRmudw1xX0LfCpiNRIRkVnRSlERkRgRiWmLIiIxpW/Iw/GWXtwjXlYXplOQnoQx/gl9Pp9l2OsjOcEZ5VpOpEAXkaiw1tIz6KGjf5iO/iG6BkYYGPbiHvEy5PGFyhkgJcFJSqKT5ARHKFittQyO+BgY9jI44iXR5SAlwUlqohOHI8xsagvtfUM0dLpp7HIzMOwJPAdGvBb3iIeBYS8t3YOc6R4c86XZqQkUpCfROTBM58AIXp8lJcFJbloi2akJOAJ1cjkN5XlpVCzJYM2SdDaVZFGYmTw3/wPDUKCLyLzpG/Kwu66D56tbef5oK/Ud7nmvgzGwJCOZjORz8edyOkhNdJKe5GJZeS5rlmRQUZhOaqKL4629HGvpo6N/iNy0HHLTEklJcNI5MEJH/zA97pHQwpvBES+7Tpzlp/saQ88uykxmS1kWN6xbws0bl5Cdmjh331u0zhStrKy0WikqEpustTR2uTnQ0M2Bhm6ONPVwvKU31PJNSXByxeo8tq/IpSAjKdDSTSQt0UlygpOkBAeGcS3xEQ+DIz6CmWWMITnBQWqCi6QEB8MeH4MjXgaGvfgmybXctESWZqWQ6Jrb4cNu9wjHWnoD338XVXWdNHa5cTkMl6/O55PXrOTyVfmzerYxZo+1tjLcPbXQRSSiWnsGufd7e9hf3wVAgtNQUZjBjvJcKpZksKkkix3luQuyDzpSslIS2L4il+0rcgH/P0pvNXbz1FtNPP1WE90DI3Pyvgp0EYmY4y29fOy7u+noH+aLt61nR3ku65ZmkOSK3fCeDmMMm0uz2VyazedvWcdcdYwo0EUkIl49cZZ7v1dFcoKTx//oMjaVZkW7SguSMQYTbgesCFCgi8gF21XTzscf3k1ZbioPf3w7pTmp0a5SXFKgi8gFqarr4J5HqliRl8aj915KbtrczeKQ89NKURGZtQMNXXz8u7spykzme/fsUJhHmQJdRGZlf30XH37wDbLTEvjBJy6hMGP+FtBIeAp0EZmx12vP8sH/eI2slAR+eM+lLM1KiXaVBPWhi8gM7TzWxh99r4qS7BR+cM+lFGWpZb5QKNBFZNrqOwa495EqVhak8727d5CfnhTtKsko6nIRkWn7h6eP4DCGhz5WqTBfgBToIjItr9We5VcHm/nja1epz3yBUqCLyJS8PsuXf3mY4qxkPnHVymhXRyahQBeRKf24qp7DTT18/rb1pCTG974sC5kCXUTOa2DYw9d/c5TK5Tm8a/PSaFdHzkOBLiLn9dO9jbT3DfM/b10XOi1IFiYFuohMylrLw7vq2FSSReXynGhXR6agQBeRSb1Sc5aa1j4+dvkKtc4XAQW6iEzq4V0nyU9P5J1b1He+GCjQRSSs02cHeK66lT/YsSzuTxxaLBToIhLWI6/W4TSGD166PNpVkWlSoIvIBN3uEX5UVc9tm5ayJFObby0WCnQRGcPj9fHpR/fhHvZy79VaFbqYaLdFERnjH56u5sVjbXz1vZu4qEQHPS8maqGLSMhjb5zmoVdO8odXlHPnjmXRro7MkAJdRACobu7hr35+kKvXFPC/blsX7erILCjQRQSA+184QZLLwTfv3IrLqWhYjPS3JiKc6XLzywNN3LljGdmpidGujsySAl1E+O4rJwH4+BUrolsRuSAKdJE41zM4wqNv1POOTUspzUmNdnXkAijQReLco6+fpm/Io5OIYoACXSSODXt8fPeVOi5bmcemUs05X+ymFejGmFuMMUeNMTXGmM+Hub/MGPO8MWafMeaAMea2yFdVRCLtVwebaO4Z5BNXl0e7KhIBUwa6McYJfAu4FdgA3GWM2TCu2F8Bj1trtwF3At+OdEVFJPJ+8PppluWmcu2awmhXRSJgOi30HUCNtbbWWjsMPAa8e1wZC2QG/jsLOBO5KorIXKhp7eWNkx3ctWMZDocOr4gF09nLpQSoH/W6AbhkXJm/AX5jjPk0kAbcGJHaicicefSNelwOwx0Xl0a7KhIh02mhh/un2457fRfwsLW2FLgN+J4xZsKzjTH3GmOqjDFVbW1tM6+tiETE4IiX/97bwNs3FlGQkRTt6kiETCfQG4CyUa9LmdilcjfwOIC19lUgGcgf/yBr7QPW2kprbWVBQcHsaiwiF+yZg810DYzwB5doA65YMp1A3w1UGGPKjTGJ+Ac9nxhX5jRwA4AxZj3+QFcTXGSB+uHrp1mel8plK/OiXRWJoCkD3VrrAe4Dfg0cwT+b5ZAx5svGmNsDxT4HfMIY8ybwKPAxa+34bhkRWQBqWnt5o06DobFoWgdcWGufBp4ed+1Lo/77MHBFZKsmInPh14daAHjf2zQYGmu0UlQkzuyu66CiMF2DoTFIgS4SR3w+y95TnVSuyIl2VWQOKNBF4sjx1j56Bj1cvDw32lWROaBAF4kju+s6ANiuFnpMUqCLxJE9pzrJT09iWa72PY9FCnSROFJ1qoPK5TkYo+mKsUiBLhInWnoGqe9wa0A0hinQReJEVV0nAJUrNCAaqxToInGi6lQHyQkONhZnTl1YFiUFukic2HOqky2l2SQ49WMfq/Q3KxIH+oc8HDrTo/7zGKdAF4kDb9Z34fVZ9Z/HOAW6SBzYdeIsDgNvW6YWeixToIvEOGstv3izkStW55OVkhDt6sgcUqCLxLi9pzup73Dz7q0l0a6KzDEFukiM+9m+RpITHLx945JoV0XmmAJdJIYNe3w8eaCJmzYUkZGs7pZYp0AXiWE7j7XRNTDC720rjnZVZB4o0EVi2M/3NZKblshVFQXRrorMAwW6SIzqGRzh2SMtvGvzUq0OjRP6WxaJUc8cbGbY4+M92zS7JV4o0EVi1G8OtVCSncLWsuxoV0XmiQJdJAYNebzsOtHOdesKdJhFHFGgi8SgPXWdDAx7uXZNYbSrIvNIgS4Sg1441kai08Flq/KiXRWZRwp0kRi082gb28tzSEtyRbsqMo8U6CIx5kyXm6MtvVyzRnPP440CXSTGvHisDYBr16r/PN4o0EVizAtH21ialUxFYXq0qyLzTIEuEkNGvD5eqWnn2rWarhiPFOgiMWTvqU56hzxco+mKcUlD4CIxoGtgmJ3H2vj+a6dwOQxXrNZ0xXikQBdZ5L753HH+5bfH8FnITUvkczev1d7ncUqBLrLIPVfdSkVhBl993ya2lGbjcKjvPF6pD11kkWvsdLO1LJtty3IU5nFOgS6yiA2OeGnvG6IkJyXaVZEFQIEusog1dQ8CUJKtQJdpBrox5hZjzFFjTI0x5vOTlHm/MeawMeaQMeaHka2miITT2OkGUAtdgGkMihpjnMC3gJuABmC3MeYJa+3hUWUqgC8AV1hrO40xmgQrMg8auwYAtdDFbzot9B1AjbW21lo7DDwGvHtcmU8A37LWdgJYa1sjW00RCaex043DQFFWcrSrIgvAdAK9BKgf9bohcG20NcAaY8wrxpjXjDG3RKqCIjK5hi43RZnJOgRagOnNQw83D8qGeU4FcC1QCrxkjLnIWts15kHG3AvcC7Bs2bIZV1ZExmrsdKv/XEKm8896A1A26nUpcCZMmV9Ya0estSeBo/gDfgxr7QPW2kprbWVBgfZqFpmuM11u/uLHb/KzfQ1jrjd2udV/LiHTCfTdQIUxptwYkwjcCTwxrszPgesAjDH5+LtgaiNZUZF45PH6ePDlk9z4jZ38ZE8D/7XrVOie12dp7h5UC11CpuxysdZ6jDH3Ab8GnMBD1tpDxpgvA1XW2icC9242xhwGvMBfWmvPzmXFReLBJx6p4vmjbVy3toC0JBfPHm7B4/Xhcjpo6RnE47OUZKdGu5qyQExrJMVa+7S1do21dpW19u8D174UCHOs32ettRustZustY/NZaUlPpxo6+PBl09GuxqTqmnt4wevn5q64CwNDHt4/mgbf3hFOQ99bDs3bVjCkMfH8dY+wN/dApqDLudoaFwWrK88Xc3fPXmYpm53tKsS1vdfO8UXf3aQnsGROXl+bVs/ANtX5GCM4aKSLADeaugGRi0qytaURfFToEtU+HyWQ2e6efDlk3zikSr+6udvYe25yVNnutz8rroFgDfruyZ7TFTVd/gX9ZwItJgj7XhrLwCrA0fJleelkZ7k4q3GQKAHWujFGhSVAG2fK1HxhZ++xY+q/Msb8tMTae8b5uYNRVwdOKn+sd31WMDlMOyr7+KWi5ZGsbbh1Xf6A72mtY9ty3Ii/vya1j5cDsPyvDQAHA7DxuLMUKA3dLrJTUskNVE/xuKnFrpExWsnz3L5qjxe/cL1vPL56ynNSeFrz1Tj81lGvD4ee+M0164pYGNJ1oQWelvvEF97ppohjzdKtQdrLfUd/hZyzRy10Gta+1iel0qi69yP6aaSLA439TDi9WnKokygQJd51z/k4dTZAS5bmcfSrBSSXE4+d/MaDp3p4am3mnjuSAutvUN88JLlbC3N4q2Gbry+c90xP9p9mu+8cILnq9ui9j2c7R/GPeL/B2WuAv14a1+ouyVoU2kWwx4fx1v6aOwcUKDLGAp0mXdHW/x9w2uLMkLXbt9SwrqiDL7+m6M8vKuO4qxkrltXyNZl2fQPe8eE5nPV/q2CfnukZX4rPkqw/zw9yUVNW+QDfdjj49TZASoKM8Zc3xQcGG3s8rfQNcNFRlGgy7w72uwP9PVLM0PXnA7D/7hlLafODvBabQd37ViG02HYUpoNwP76TgDO9g2xv74Ll8Pwu+rWMS33+VQfmGFyVUU+pzsGGByJbPfPqbP9eH12Qgt9RWBg9MVj7QyO+NRClzEU6DLvqpt6SEt0Tgij69YWsmNFLi6H4QPb/btNlOenkZnsYn+9fyBw57E2rIW7ryyno3+Yfac7573+cK6Ffu3aAqw9N8UwUoJzzccHenBg9Pmj/t9S1EKX0RToMu+ONPeytihjwvmXxhj+5c6tPHL3Dgozk0PXtpRlsz8wMPq76lYKMpL4k+tWk+A0PDtJt4vPZ3n0jdOh4I204AyTLWX+3yAi3e1S09qHMbCqIH3Cvc2lWQwM+38jUAtdRlOgy5wa8njxeH2h19Zajjb3sm5Ud8toxdkpXL4qf8y1bWXZHGvppWdwhJ3H/Mvgs1ISuHRlHr89HD7Qnz/ayhd++hbv/NeXeeHozLfn/+3hFp452Dzp/YbOAcpyUijPT8NhoCYwLhApNa19lGSnkJLonHAvuMAIoFQtdBlFgS5z6n3f2cVf/+Jg6HVzzyDd7hHWFWWc56vG2lKWjddneWRXHb2DHq5f5z8Q68b1SzjR1k9tmNbxw7vqWJKZRHF2Ch9/eDf/+txxfDPob//6b47ytWeqJ71f3zFAaW4qSS4ny3JTI95CDzfDJSg4MJqW6CQrJSGi7yuLmwJd5syps/0cbOzhl282hQYNqwMDouuKwrfQwwl2azzwYi0JTsOVFf7FRzes9wf7c0fGtsCPt/Ty0vF2PnLZCn76x5dz+5Zi/unZY/zny9PbAHTY46OmtY+T7f30hlnW7/VZGrvclOX4N8VaXZgR0amLXp+ltq2PikkCPTgwWpKTgjHhjiuQeKVAlzmz85h/nnjfkIeXjrcDUN00ccriVPLTkyjNSaFn0MOO8lzSk/wrI0tzUlm/NHNCP/p/vVpHosvBXTuWkZLo5F8+sJUrV+fz4MsnGfb4wrzDWCfa+vAEWvNHmiZ2pbT0DDLitZTl+rs7Vhemc7K9f0zX0kw9c7CJzv5hwL9Hy5DHN2kL3eEw3LRhCZeU5836/SQ2KdBlzuw82kZZbgpZKQn86q0mAKqbeyjOSp5xV8HWQCv9urVjzx+/aX0hVXUdoX1Nut0j/PeeRt6ztZjctETAP7B6z1XltPQM8dRb489mmSg4rRLg0JnuCfeDA63nWujpjHgtp2Y5AHugoYtPfn8vf/aj/VhrJ+zhEs4/f2Arf/eei2b1fhK7FOgyJwZHvOw6cZbr1xZy04YlPHu4hSGP97wDoudTuTwHY+CG9UvGXL99azEJTgfv+teX+e3hFh7fXY97xMtHL18xptw1awqoKEznP186OWYTsHCONPeQ6HSQm5bIoTM9E+4H56CX5foDPdg1Mttulx/t9u9ps/NYGz+uagg9Z3XB9H+LEQEFusyRqrpO3CNerllbwDs2LaV3yMPz1W3UtPbNqLsl6K5LlvGLT11BeX7amOurCzN48tNXUpSZzD2PVPGNZ4+xozyXjcVZY8oZY7j7ynIOnenhtdqO877X0eZeVhWmc1FJVvhA7xjAGCgObFu76gIC3T3s5Yn9Z3jP1mIuKc/l7548zMs17RRkJJGVqgFPmRkFusyJF462kuhycOnKPK5YnU9Gsot/e/44Hp+d0QyXoCSXk82BVaPjVSzJ4Gefupw/unolw14fn7xmZdhy79lWQl5aIv/50vkHR6ubellflMHG4kyOt/RO2ASsvnOAosxkklz+KYXpSS6WZiVPGegjXh+/2N/IwLAndO3pt5roHfJw545l/OMdW/D4LC8db2d1mPnnIlNRoMuc2HmsjUvKc0lNdJHocnDThiUcbPS3dtfPostlKkkuJ1+4bT2H/vbtXL9uSdgyyQlOPnzZcp6rbuXEJNMMuwaGae4ZZN3SDC4qzsLjsxxrHlu2oePcDJeg1YXpUwb69187xWce289f/vhAqNvnR1X1rMhL5ZLyXJblpfKF29YBULFEgS4zp0CXiGvscnO8tY9rAnubA9wW2M88wWkmdJtEUnLCxIU4o33o0uUkuRz8/VNHws5LD06rXFuUycZi/z884wdG6zsHKM0du6AnGOiT7eniHvbyredPkJ2awFNvNfHQK3XUtvXxxskO3r+9LDT98EOXLOfPb1wT2vpAZCYU6BJxO4/6pyteu/ZcoF+1Jp/0JBerCzNIcEbvY5efnsT/um09v6tu5dsv1Ey4X90U+C2iKINluamkJ7nG9KMPebw09wxSOq6FfvWaAtwjXt73nV1htxt45NU62vuG+I+PVHLzhiV85ekj/M0vD+N0GO54W2monMNh+MyNFRPGAESmQ4EuEbfzWCsl2Slj9iFJcjn50js38KnrVkWxZn4fuWw5797qX2z00vGxe6ofbeklJzWBgowkHA7DhuLMMS30M12DWAtlORM3Fnvwo5Wc7hjgXf/28pjn9g15uH/nCa5eU8D2Fbl8/f1bKMtN5cVjbVy3tjC0b43IhVKgS0RZa3n1xFmuqsifsIrx/dvLeOfm4ijV7BxjDF957yYqCtP500f3heawg38h0bqizFDdNxZncqSpN7RNb2gOem7qhOfesH4JT9x3JYUZSXz4wTf4s8f2Ud8xwMOvnKRzYITP3rQGgMzkBL7zobexujCde68OP4ArMhsKdImotr4hegY9czLwGUmpiS7u/9DFjHgtn3vcv6DH57Mca+ll3dJzs3A2FmfhHvFyst0/4Bk8RzRcoIN/u9+f/ckV/PG1q3jmUDPX/9MLfOv5E9y4vjC0OAr8Wx/89rPXsKM8dw6/S4k3CnSJqOC+4HM58BkpKwvS+eI71vNabQc/3tNAfecAA8PeMdMqLyoJDoz24PNZ9tR1kuA0FJ2nmyQtycX/vGUdL/zFddxxcSnJCQ4+d/PaOf9+RBToMmtHm3sZGbd/STDQVxYs/EAH+EBlGTtW5PL3Tx3h5Rr/fjOjNw5bVZBOosvBs4dbuPM/XuOn+xq5fUsJTsfUm2IVZSXzlfduZt+Xbl7wv7FIbFCgy6x09g/zjm++xA9fPz3mem1bH8kJDoqzFsc+3Q6H4R/eexHuYS//58kjGANrlpxroSc4HawvyuDJA00caerhH+/YzNd/f3MUaywyOQW6zEpDpxuPz4ZOEgqqbe9nRV7ahNOIFrLVhRn8yXWrcI94WZGXNuFQiQ9espzf21bCs39+Db9fWaYta2XBckW7ArI4NfcMAnB43F4ntW19i3IO9R9fu4pnDjazuXRi3d+/vYz3a6GPLAIKdJmVYKDXtPlXRyYnOBn2+KjvdPOuLdGfmjhTSS4nT9x3Ja5F9JuFyHjqcpFZae72z932Bqb6AZzu6Mfrs4tmQHS8RJdjUXUViYynQJdZae4eIsHpD7/g0vgTwRku+dpYSiQaFOgyKy09g2xYmklGkivUj77YpiyKxBr1ocusNPcMsrognaQEZ2ivk9q2PgoykshI1sEMItGgFrrMSnP3IEVZyWxYmkl1s3+vk9r2flYughWiIrFKgS4z1jfkoW/Iw5LMZDYWZzIw7KXubD+1bX2s1Ek7IlGjLheZseZu/5TFpVnJoZN1dtW00zkwwir1n4tEjVroMmMtgTnoSzKTqSjMIMFp+OWBJkADoiLRNK1AN8bcYow5aoypMcZ8/jzl7jDGWGNMZeSqKAtNU6CFXpSVTKLLQUVhBrvrOgBNWRSJpikD3RjjBL4F3ApsAO4yxmwIUy4D+FPg9UhXUhaWYAs9uIXsxuJMrPWfF1qaszg25RKJRdNpoe8Aaqy1tdbaYeAx4N1hyv0d8H+BwQjWT6KgvmOAOx94lc7+4bD3m7sHyUpJCG1itSFwmPLyvDRcUTwvVCTeTeenrwSoH/W6IXAtxBizDSiz1j4ZwbpJlLx64iyv1Xaw51Rn2PvNPYNjDngIbsalKYsi0TWdQA+3uYUN3TTGAfwz8LkpH2TMvcaYKmNMVVtb21TFJUoaAses1QaOXRuvuXuQJVnnAn390gycDhOa8SIi0TGdQG8ARu8dWgqcGfU6A7gIeMEYUwdcCjwRbmDUWvuAtbbSWltZUFAw+1rLnGro9G+8FVzKP56/hZ4Uep2RnMAP7rmEe67Ugcci0TSdeei7gQpjTDnQCNwJ/EHwprW2G8gPvjbGvAD8hbW2KrJVlfkSCvT2iYE+4vXR3jdE0bgTiS5dmTcvdRORyU3ZQrfWeoD7gF8DR4DHrbWHjDFfNsbcPtcVlPnX2DV5C72tdwhrOe8hySISHdNaKWqtfRp4ety1L01S9toLr5ZEy4jXR1O3m7REJ+19Q/QMjpA5arOtc3PQkyZ7hIhEieaYyRjN3YP4LFy2yt+FcnJcK/3cHHTNNxdZaBToMkZ9YIbL1Wv8g9bjZ7o0j1olKiILiwJdxmgMDIhevioPp8NM6Edv6Rkk0eUgJ1V7nossNAp0GaOh040xsCw3jbKclAmB3tQ9yJLMJIzR2ZsiC40CXcZo6HRTlOnfdKs8P23C1MXmnkGWqv9cZEFSoMsYDZ0DlGT7A3tlQTon2/vw+UILg2npGbtKVEQWDgW6jNHQ6Q7tmLiyII3BER9NgZkt1lr/0XOZmrIoshAp0CXE4/XR3DNIaU4qAOWBzbZq2/wzXbrdIwx5fBNWiYrIwqBAl5DmnkG8Phtqoa8KnA96MtCPvvOYf0M1nUoksjAp0CUkuIdLSSDQCzOSSEt0UtvWz4jXxz8/e4x1RRlcU6GN1UQWIgW6hAQDPdjlYoxhZUE6J9r6+O89DdSdHeAvbl6Lw6EpiyIL0bT2cpH4EFxUVJx9bhZLeX4ab5zs4P+1HmfbsmxuWF8YreqJyBTUQpeQhs4BlmQmkeRyhq6tLEijuWeQpu5B/vLmtVpQJLKAKdAlpKHTHZqDHrQyMDB6xeo8Ll+dH+7LRGSBUKBLSEPXQKj/POji5TmsLEjjC7euj1KtRGS61IcuAHh9lqauQd61eWwLvSQ7hd997troVEpEZkQtdAH8S/o9Phuasigii48CXYCJUxZFZPFRoAsAJwLL+0vVQhdZtBToAsAT+8+wLDeVlfla1i+yWCnQhfqOAV6tPcsdF5dqnrnIIqZAF366txGA976tJMo1EZELoUCPcz6f5Sd767l8VZ4GREUWOQV6nNtd10F9h5vfryyNdlVE5AIp0OPcT/Y0kJ7k4u0bi6JdFRG5QAr0ReiT39vD156pxlo7deHz6B/y8NRbTbxj01JSE7VoWGSx00/xItPtHuGZQ80AZKck8EfXrJr1s5460MTAsJc71N0iEhPUQl/A9p7uZGDYM+baWw3dAKxZks5XflXNz/c1zurZXp/l/p0nWFeUQeXynAuuq4hEnwJ9gWrtHeSO7+zi33fWjrn+ZkMXAN+/5xIuXZnLX/7kTXadaJ/x8588cIba9n4+c0OF5p6LxAgF+gJVVdeJzzIhrN+s76I8P43CjGQe+EglBelJPPBi7SRP8atr7w8d8Az+qYr/9rsa1ixJ12CoSAxRoC9Qu+s6ANhf34V72Bu6fqChm82lWQBkJiewqTQrtLFWOPUdA9xx/6t89KE3+OZzx7HW8syhZo639nHf9RU6H1QkhmhQdIHac6qT1EQnA8Neqk51cFVFAS09gzT3DLKlNDtUrjQnlRePtWOtndB10tE/zEcfeoMRr49bNhbxjWeP0dE/zGu1Z1lZkMY7Ni2d729LROaQWugLUP+Qh0NnevV8FRMAAA13SURBVPjA9jKcDsOrJ84C/u4WgC1lWaGyJdkpuEe8dPQPj3mGe9jL3f+1m8YuNw9+tJJvf/Bt3HNlOQ/vqqO6uZdPX78ap1rnIjFFLfQFaH99F16f5dq1heyv7+LV2kCgN3ThdBg2Fp8L9OB2t41dbvLSk0LXv/ZMNW/Wd/HtD15M5YpcAL74jvUszU6hqq6Dd20unsfvSETmg1roC9Duug6MgW3LsrlsZR4HGrrpH/LwZn0364oySE5whsoGTxga34++93QnV6zO55aLzg16GmO4+8pyvvOhi3E59VcvEmv0U70AVdV1sq4ok8zkBC5blYfXZ3mjroMDDV1sHtV/DlCa7d9Qq3FUoFtrqW3rZ1VB+rzWW0SiS4G+wHi8Pvae7mT7Cv9in8rluSQ4DY+9cZqeQQ9bR/WfA2SmuMhIctHQORC61tY3RN+Qh3IdViESV6YV6MaYW4wxR40xNcaYz4e5/1ljzGFjzAFjzHPGmOWRr2p8qG7uZWDYG+r3Tkl0srUsm98cbgGY0EI3xlCSk0Jj17kW+sm2fgAFukicmTLQjTFO4FvArcAG4C5jzIZxxfYBldbazcBPgP8b6YouZPUdA/yuuiUizwrOPw+20AEuW5mHtZCS4KSicGI3SmlOypg+9JPtCnSReDSdFvoOoMZaW2utHQYeA949uoC19nlrbfB3/teAuNntyVrLfT/cyx8+XMWBwLL8C1FV10lJdgpLs84d1nzpqjwALirJDDuYWZKdMqYPvba9n0SXg5JsHfgsEk+mE+glQP2o1w2Ba5O5G/jVhVQq2j71w73cv/PEtMo+d6SVNxu6cToMX/7l4Qva0tZay+66DipXjN0s623LckhPcrE90A0zXklOCr1DHrrdIwDUtvVTnpemVaAicWY6gR4uFcKmljHmQ0Al8I+T3L/XGFNljKlqa2sLVyTqPF4fvznUzO+OtE5Z1uezfOPZY6zIS+Vvb99I1alOnjzQFLbsmS43X/zZW3QPjEz6vMYuN629Q1w8bvfD5AQnv/rMVXz6+oqwXxc8Oi44MHqyvU/dLSJxaDqB3gCUjXpdCpwZX8gYcyPwReB2a+1QuAdZax+w1lZaaysLCgpmU985d7pjgBGvpaatb8qyvz7UzOGmHj5zYwV37VjGhqWZfPVX1QyOeCeUffLAGX7w+mk++/h+fL7wrfjjLf733LA0c8K9stxUUhKdE64Doa6Vxk43Hq+P0x0DlBco0EXizXQCfTdQYYwpN8YkAncCT4wuYIzZBvw7/jCfumm7gNW0+kO1o394wnL60bw+yz//9hirC9O5fUsJTofhS+/aQGOXO+zuh8FumeeqW/n2CzVhn3mspReAisKMGdV59GrRxi43I16rFrpIHJoy0K21HuA+4NfAEeBxa+0hY8yXjTG3B4r9I5AO/NgYs98Y88Qkj1vwRrfMa8/TSv/lm2c41tLHn91YEdoT5dKVedx6URH37zwxoZV+oKGLt29cwu1bivmnZ4/x0vGJXU7HW/sozEgiKzVhRnXOTUskOcFBQ6eb2sCUxVVqoYvEnWnNQ7fWPm2tXWOtXWWt/fvAtS9Za58I/PeN1tol1tqtgT+3n/+JC9eJ1n4SAzNJTkwS6Eeaevjrnx9kU0kWt100dsfC368sZWDYy97TnaFrHf3D1He42VyazVfft4mKwnQ+89h+zvaN7Zk63tJLxZKZr+40xoRmutSGpixqlahIvNFK0XFq2vq4eHkOSS5HqPtltMYuNx/77hukJbn49w9fPGEmSeWKXBwGXqvtCF0LTmfcXJpFaqKLr75vMx39w7x0/NzhFdZajrf2zbi7Jag0J5WGrgFOtveRlZJAzgxb+SKy+MVVoHe7R9jyt7/hmYPhZ6JYaznR2seaJemU56dxItB9EdQ14N9ffGDYy8N/uJ3iMPO8M5MT2FSSxWuBLW/BfyiFMbCpxL9sf3NJFqmJTvaNasU3drkZGPbOqoUO/qmLjZ1uTrb3U56fpmPlROJQXAX6vtOddLtHePZw+HHblh7/HiirCtNZVZg+ocvlS784xOmzAzzw4UrWFU2ciRJ06cq8MScNHWjoYmV+GhnJ/lazy+lgc2kW++rPLUQ6HvhtYPYt9BQ6B0Y4dKaHleo/F4lLcRXo+wMBuudUR9j7wS6W1QXprCpIp75jIDS46fH6eL66lfddXMJlgZWbk7l0ZR7DXh/7TndirWV/ffeYU4YAti3L4fCZntDzj4dmuMyyhR74baFrYISVmuEiEpfiKtCDJ/7UnR2grXfiVPlgi3x1YTqrC9PxWag76+92OdDYTe+QhytW50/5PpUrcgL96Gdp6h6kvW8odA5o0LaybDw+y8HGbsA/Bz0/PYmctMRZfW/BqYugAVGReBU3ge5vKXexOtACDtdKr2ntIyPZRUFGUmja34lWf6DvqvEPYF6+aupAzwj2o9d2hAZEt5SNbaFvXeZ/ve+0//7xQN/9bAVXi4I25RKJV3ET6PUdbjoHRvjgJctIdDmoquucUKamtY/VhekYY1iZn44x51rtL9e0s7E4k9xptqCD/eivn+zA5TCsH7f6szAjmdKcFPbV+7tlalr7Zt3dAlCQnhSabrkiP3WK0iISi+Im0PfV+wN8+4pctpRmUXUqTKC39YVO+UlJdFKSncKJtj7cw172nuqaVndLULAf/SdVDaxbOvbYuKBty3LYd7qLpu5B+oY8VCyZ3YAogMNhKM5OpjgrmdREHRUrEo/iJtDfrO8mOcHB2qIMLl6ey6Ez3aFZKOCf0tjWOxTqkgFYVZBOTWsfu+s6GPb6ZhTowX703iHPhEMpgraVZdPUPRhaNXohLXTw78q4ozz8jowiEvviJ9AburioOIsEp4PK5TmMeC1vjtq/PDQgWjA20Gvb+nnpeBuJTseYQyemEuxHB9gybkA0aFugH/3xqgYA1lxACx3gGx/Yyr/cue2CniEii1dcBPqI18fBxm62BgYmg9vT7hnV7RKasji6hV6YhnvEy8/3n2HbsuwZd2UED6aYrIW+oTiTRKeDPac6yU9PnPUMFxERgLjobD3a3MuQxxeaaZKTlsjqwnSq6s7NdDnR2kei0zFm+l+wtd7WO8RHLp35MakfuWwF6Yku1k7S8k5yObmoJJO9p7tmvaBIRCQoLlrowRWZW0dNHaxcnsOeU52hvclrWv2HQow+4m3VqNb6FRXT7z8PKslO4dM3VJz35KBty/y/Lcx2yb+ISNCia6F39A/T3hf2/IwxEp0OluelYozhzfou8tISx7S+L16ew2O769l7upPW3iHebOjikvKxK0Dz0hLJSknA57NsLgnfD36hgv3oFzLDRUQEFmGg/7iqnq/8qnpaZSuX5/BnN65hf30XW8uyx2xYVRk4n/OO+18FIDPZxW2bxm6Fa4zhkvJcMpITwh7OHAlXrS7g+nWFXLtmYZ7gJCKLh7mQQ40vRGVlpa2qqprx151o66O6qXfKck3dbh58+SRN3YMAfPamNfzpDefO5LTW8n+eOkKiy8H16wrZVpY9aWhba7V7oYgsCMaYPdbaynD3Fl0LfVVg46zp+PBly3m8qoGf7W3g1ouKxtwzxvDX79wwrecozEVkMVh0gT4TSS4nH750OR+exQwVEZHFJi5muYiIxAMFuohIjFCgi4jECAW6iEiMUKCLiMQIBbqISIxQoIuIxAgFuohIjIja0n9jTBvQBXRPUiTrPPfygfa5qNccO9/3tJDf60KeNdOvnW756ZSbqsxk9/X5mt/3irfPF1zYZ2y5tTb85k/W2qj9AR6Y5b2qaNZ7Lr7fhfxeF/KsmX7tdMtPp9xUZSa7r8/X/L5XvH2+Avfm5DMW7S6XX87y3mI1n99TJN/rQp4106+dbvnplJuqTKx9xvT5ilz5Rfn5ilqXy4UwxlTZSXYbE7lQ+nzJXJurz1i0W+iz9UC0KyAxTZ8vmWtz8hlblC10ERGZaLG20EVEZBwFuohIjFCgi4jEiJgLdGPMSmPMg8aYn0S7LhIbjDFpxpj/Msb8hzHmg9Guj8SWSGbWggp0Y8xDxphWY8zBcddvMcYcNcbUGGM+f75nWGtrrbV3z21NZbGb4WftvcBPrLWfAG6f98rKojOTz1ckM2tBBTrwMHDL6AvGGCfwLeBWYANwlzFmgzFmkzHmyXF/Cue/yrJIPcw0P2tAKVAfKOadxzrK4vUw0/98RcyCOiTaWvuiMWbFuMs7gBprbS2AMeYx4N3W2q8A75zfGkqsmMlnDWjAH+r7WXiNIFmAZvj5Ohyp910MH84SzrWOwP/DVTJZYWNMnjHmfmCbMeYLc105iSmTfdZ+CrzPGPMdYm+7AJk/YT9fkcysBdVCn4QJc23S1VDW2rPAJ+euOhLDwn7WrLX9wMfnuzIScyb7fEUssxZDC70BKBv1uhQ4E6W6SGzTZ03m0px/vhZDoO8GKowx5caYROBO4Iko10likz5rMpfm/PO1oALdGPMo8Cqw1hjTYIy521rrAe4Dfg0cAR631h6KZj1l8dNnTeZStD5f2pxLRCRGLKgWuoiIzJ4CXUQkRijQRURihAJdRCRGKNBFRGKEAl1EJEYo0EVEYoQCXUQkRijQRURixP8HN/H5yIM8K1wAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.semilogx(bandwidths, grid.cv_results_['mean_test_score'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 0.9682804674457429\n"
     ]
    }
   ],
   "source": [
    "print('accuracy:', grid.best_score_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们发现，这个不是很朴素的贝叶斯分类器的交叉检验准确率达到了 96%，而朴素贝叶斯\n",
    "分类器的准确率仅有 80%："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.model_selection import cross_val_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\Anaconda3\\lib\\site-packages\\sklearn\\model_selection\\_split.py:1978: FutureWarning: The default value of cv will change from 3 to 5 in version 0.22. Specify it explicitly to silence this warning.\n",
      "  warnings.warn(CV_WARNING, FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.8186003803550138"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_val_score(GaussianNB(), digits.data, digits.target).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这种生成分类器的好处是结果很容易解释：我们不仅得到了每个未知样本的一个带概率的\n",
    "分类结果，而且还得到了一个可以对比的数据点分布全模型（full model）。如果需要的话，这个分类器还可以提供一个直观的可视化观察窗口，而 SVM 和随机森林这样的算法却难以实现这个功能。  \n",
    "  \n",
    "如果你还想进一步改进这个 KDE 分类模型，那么可以这么做。  \n",
    "• 允许每一个类的带宽各不相同。  \n",
    "• 不用预测值优化带宽，而是基于训练数据中每一个类生成模型的似然估计值优化带宽（即使用 KernelDensity 的值，而不使用预测的准确值）。  \n",
    "  \n",
    "最后，如果你希望构建自己的评估器，那么也可以用高斯混合模型代替 KDE 来构建一个\n",
    "类似的贝叶斯分类器。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### This section is accomplished!!!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
